The present invention relates to a method and an apparatus for synchronized play back of audio-video signals, which have a high performance when particularly plural digitally compressed audio data and digitally compressed video data are decoded and played back.
Generally, an apparatus for synchronized play back of audio-video signals reads digitally compressed audio-video (AV) signals recorded in a recording medium. such as a CD-ROM or a hard disk (HD) and plays back on a CRT monitor and to a speaker by decompressing the signals.
However, at a conventional apparatus for synchronized play back of audio-video signals, the performance of a clock is not sufficient, consequently, there is a problem that sufficient resolution applying for synchronizing control can not be obtained. In order to solve this problem, Japanese Patent Application Laid-Open No. HEI 10-136308 discloses an apparatus for synchronized play back of audio-video signals.
In this application, reference time is obtained from the amount of data of played back audio data (for example, pulse code modulation (PCM) data), and video playing back is made to synchronize with this reference time (for example, by frame dropping of video data), and synchronized play back of audio-video signals is performed.
In this apparatus, as mentioned in this application, audio signals can be processed as the first priority, consequently the audio data do not become intermittent. Therefore, the play back without abnormal feeling can be performed, and when frame dropping is performed, the frame dropping is performed in the order of the frame whose priority is high, as a result, video signals can be smoothly played back.
However, at conventional apparatuses for synchronized play back of audio-video signals, a case that plural compressed audio data are multiplexed is not considered. For example, at the conventional technology of the mentioned above application, at the play back of the compressed AV data whose plural audio data are multiplexed, there is a case that audio data are changed over. In this case, when the compressed multiplexing AV data to be separated are merely changed over, generally, the AV separation from the compressed AV data is sequentially performed. However there is a problem that the compressed audio data applied the AV separation, which should be essentially played back at the same time at before and after the changing over, are overlapped, or the separated compressed audio data to be played back at designated time are lost.
FIGS. 1 and 2 are conceptual diagrams showing a series of compressed multiplexing AV data and a series of compressed audio data applied AV separation, at the conventional apparatus for synchronized play back of audio-video signals. Referring to FIGS. 1 and 2, the problem mentioned above is explained in detail. In FIGS. 1 and 2, in Audio NT, N means the number of audio and T means time.
For example, in FIG. 1, when the compressed multiplexing AV data are changed over from audio (0) to (1) at point A, at the separated compressed audio data, the audio data at time (0) and (1) are overlapped. In FIG. 2, when the compressed multiplexing AV data are changed over from audio (1) to (0) at point B, at the separated compressed audio data, the audio data at time (2) and (3) are lost.
As mentioned above, at the conventional apparatus in which reference time is obtained from the amount of cumulative data (for example, PCM data) outputted from the starting of play back, the reference time after the audio data are changed over has time lag or lead, and the synchronized play back after the audio data are changed over is not secured.
It is therefore an object of the present invention to provide a method and an apparatus for synchronized play back of audio-video signals in which time lag or lead of the reference time after the audio data are changed over can be avoided and the performance of the synchronized play back after the audio data are changed over can be increased, even in a case that plural digitally compressed audio data are separated and played back.
According to a first aspect of the present invention, for achieving the objects mentioned above, a method for synchronized play back of audio-video signals, which decodes and plays back plural digitally compressed audio data and digitally compressed video data, when said plural digitally compressed audio data are changed over and one of said plural digitally compressed audio data is selected and played back, provides the steps of obtaining reference time based on the amount of played back data of said changed over decoded audio data, and playing back said changed over decoded audio data synchronizing with video data decoded said digitally compressed video data, based on said reference time.
According to a second aspect of the present invention, in the first aspect, said reference time T based on the amount of played back data of said changed over decoded audio data is calculated by an equation, T=T2+T1xe2x88x92T3, using the following T1, T2 and T3, decoding start time T1 for said changed over audio data to be decoded, played back time T2 of decoded audio data calculated by the amount of the played back data of said changed over decoded audio data, and decoding start time T3 for said video data to be decoded.
According to a third aspect of the present invention, in the first aspect, when said changed over decoded audio data are played back synchronizing with said video data decoded digitally compressed video data, based on said reference time, the present invention further provides the step of comparing time that said reference time T is added to said decoding start time T3 for said video data to be decoded with decoding time of the oldest inputted frame to be decoded in frames of undecoded video data in said digitally compressed video data. When said added time is prior time to said decoding time of said oldest inputted frame to be decoded, by judging a preceding state whose preceding time is time from said added time to said decoding time of said oldest inputted frame, the present invention provides the step of weighting for decoding for said video data corresponding to said preceding time. And when said added time is posterior time to said decoding time of said oldest inputted frame to be decoded, by judging a delaying state whose delaying time is time from said decoding time of the oldest inputted frame to said added time, the present invention provides the step of frame dropping of number of frames corresponding to said delaying time for said video data.
According to a fourth aspect of the present invention, in the third aspect, said frame dropping of the number of frames corresponding to said delaying time for said video data makes frames to be dropped give the order of priority and performs the frame dropping from the frames having high priority.
According to a fifth aspect of the present invention, an apparatus for synchronized play back of audio-video signals provides an audio-video (AV) separating means, to which compressed multiplexing data of digitally compressed video data and plural digitally compressed audio data are inputted, and separates the inputted data into the digitally compressed video data and instructed audio data corresponding to an index (ID) instructed from an external input through an audio changing over means in said plural digitally compressed audio data, a video decoder, in which said digitally compressed video data separated at said AV separating means are decoded, and outputs the decoded video data, an audio decoder, in which said instructed audio data separated at said AV separating means are decoded, and outputs the decoded audio data, an audio play back controlling means in which playing back of said decoded audio data is controlled, an audio counter which calculates the amount of played back data of said decoded audio data played back at said audio play back controlling means, a clock generating means which calculates reference time based on said amount of played back data calculated at said audio counter, decoding start time of said video data at said video decoder, and decoding start time of said instructed audio data at said audio decoder, and a synchronizing controlling means which makes said decoded video data and said decoded audio data synchronize for playing back by controlling the operation of said video decoder based on said reference time.
According to a sixth aspect of the present invention, in the fifth aspect, at said clock generating means, reference time T based on the amount of played back data calculated at said audio counter, the decoding start time of said video data at said video decoder, and the decoding start time of said instructed audio data at said audio decoder is calculated by an equation, T=T2+T1xe2x88x92T3, using the following T1, T2 and T3, decoding start time T1 of said instructed audio data corresponding to said ID instructed by said external input, played back time T2 of decoded audio data calculated by the amount of the played back data of said instructed audio data corresponding to said ID instructed by said external input, and decoding start time T3 for said video data to be decoded.
According to a seventh aspect of the present invention, in the fifth aspect, when said decoded audio data of said instructed audio data are played back synchronizing with said video data of said decoded digitally compressed video data, said synchronizing controlling means compares the time that said reference time T is added to said decoding start time T3 of said video data to be decoded with the decoding time of the oldest inputted frame to be decoded in frames of undecoded video data, when said added time is prior time to said decoding time of said oldest inputted frame, by judging a preceding state whose preceding time is time from said added time to the decoding time of said oldest inputted frame, said synchronizing controlling means performs weighting for said video data corresponding to said preceding time, and when said added time is posterior time to said decoding time of said oldest inputted frame, by judging a delaying state whose delaying time is time from said decoding time of said oldest inputted frame to said added time, said synchronizing controlling means performs frame dropping for said video data corresponding to said delaying time.
According to a eighth aspect of the present invention, in the seventh aspect, at said synchronizing controlling means, said frame dropping of the number of frames corresponding to said delaying time for said video data makes frames to be dropped give the order of priority and performs the frame dropping from the frames having high priority.
According to a ninth aspect of the present invention, in the fifth aspect, said audio changing over means inputs said ID to said AV separating means in order to instruct said audio data, and said audio counter, based on a signal outputted from said audio changing over means, makes the calculated result of the amount of played back audio data of said decoded audio data played back at said audio, play back controlling means reset.
According to the present invention, an apparatus for synchronized play back of audio-video signals of the present invention decodes digitally compressed audio data and digitally compressed video data and plays back the decoded audio data synchronized with decoded video data. And the apparatus of the present invention controls the synchronization based on reference time obtained from the amount of played back audio data of decoded audio data. When compressed AV data with plural compressed multiplexing audio data are played back and the plural compressed audio data are changed over, and even at the case that discontinuity occurs on the time base of the compressed audio data to be separated, corresponding to the reference time obtained from the amount of played back data of the decoded audio data, the audio data can be played back synchronizing with the video data by the same synchronizing control method before the audio data are changed over.
An operation of an apparatus for synchronized play back of audio-video signals of the present invention is explained. Pulse code modulation (PCM) data stored in a PCM buffer memory are played back at an audio play back controller and outputted to a speaker. The amount of PCM data (decoded audio data) played back at the audio play back controller is counted at an audio counter and reference time is calculated from this cumulative amount of played back data at a clock generator. Based on this reference time, a synchronizing controller judges whether a delaying state or a preceding state at a video decoder and controls the synchronizing operation.
At the case that the compressed audio data recorded in a data recording equipment are plural multiplexing data, when the changing over of audio data is instructed from an external input, an audio changing over controller makes the audio counter reset the amount of played back audio data before the audio data are changed over. And the clock generator corrects the calculating reference time.
By this correction of the reference time, even after the audio data are changed over, the clock generator can calculate the reference time continuing to before the audio data are changed over with the amount of the played back audio data. And the synchronizing controller can make the video decoder perform the same synchronizing play back before the audio data are changed over.